Analyze BBA models fitted to simulated anti-elite codings datasets varying n and n[i]

Posterior classification uncertainty

Here, I define posterior classification uncertainty (PCU) as the standard deviation in posterior estimates across chains and iterations: \[ \text{SD}(\mathbf c_i) = \sqrt\frac{\sum_{t=1}^T (c_{it} - \bar{c}_i)^2}{T-1}, \] where \(t\) indexes the \(t^{th}\) estimate and here \(T = 1000 \times 3\) (iterations times chains). Note that the theoretical maximum of PCU is reached if an item is estimated to be a member of the positive class exactly half T/2 times, and this maximum approaches .5 as \(T \to \infty\).

A first view at the distributions ofUs for different values of \(n\) and \(n_i\) illustrates that our posterior classifications are comparatively uncertain if we aggregate only few judgments per item. As \(n_i\) increases, however, the PCU of most items is reduced substantially, and in the extreme case of \(n_i = 10\) is reduced to neglible levels in virtually all itmes. We can also see that increasing \(n\) contributes only little to change this pattern.

These results are summarized by the following figure that plots the change in mean PCU and the standard deviation inUs for different combinations of \(n\) and \(n_i\).

While we see now more clearly that differences in \(n\) make no significant difference, neither for changes in the mean PCU (left-hand panel) nor in the standard deviation of PCUs (right-hand panel), for all values of \(n\) there occurs some reduction in mean PCU values as \(n_i\) is increased. Similarly, The standard deviation in PCUs is lower for higher values of \(n_i\). Taken together, we thus see that both average PCU values and their variability decreases as \(n_i\) is increased.

The above plot says little about the significance of the reductions in mean PCUs, however. Therefore, the below table reports the proportion of items whose PCU is increased if aggregating \(n_i + l\) instead of \(n_i\) judgments for \(l \in (1, \ldots, 7)\) (displayed in columns 3–9). The proportion of items with positive PCU change from \(n_i\) to \(n_i + l\) can be interpreted as a significance test: if less than 5% of items see increase, we are 95% confident that the change in PCU induced by collecting and aggregating an additional \(l\) judgments leads to an average decrease inUs.

So, for instance, for \(n = 200\), if we aggregate four instead of three judgments per items, this leads to a decrease in PCU in only about 94.5% of cases. We are 98.5% (97.5%) certain, however, that collecting seven (eight) judgments instead of three judgmentes per item results in an average decrease in item-levelU, which meets the 5% confidence level criterion. In order to recude average PCU levels, it thus seems advisable in most cases to collect multiple judgments per item (i.e., increase \(n_i\)) rather than judgments for new items (i.e., increase \(n\)).

Another point underlining the added value of collecting multiple judgments per item while holding \(n\) constant is illustrated by the following figure that separates items with positive average PCU change aggregated over all possible judgment increases from \(n_i\) to \(n_i +1\) for \(n_i \in \{3, \ldots, 9\}\) from all other items. These items are important as average PCU increase characterizes items that the models tend to classify erroneously with high levels of posterior certainty when aggregating only few judgments per item and whose classification becomes on average less certain if we add judgments.

We see indeed that if we collect only few judgments per item (i.e., low values fo \(n_i\)), items that are characterized by large PCU when aggregating many judgments per item are rather proportionally distributed across the entireU-value range. Thus, when collecting only few judgments per item, we are not able to separate items with an average increase in PCU as \(n_i\) is increased from other, apprently less problematic items. Indeed, such separation is usually only possible for values of \(n_i > 5\) (significantly, the size of \(n\) makes no apparent difference in this regard).

Especially worrying is that the uncertainty about posterior classification of some of the items that the model classifies quiet confidently based on relatively few judgments per items increases as we collect more judgments for these items: Indeed, for these items we observe a radical increase in PCU as \(n_i\) is increased. If we wouldn’t have collected more judgments for these items, we would be lead to err in believing that we know their true classes quiet confidently.

Note, however, that the absolute proportion of these items is rather small overall, as the following figure documents.

Indeed, this is good news: increasing \(n_i\) leads to average reduction in PCU for the vast majority of items apparently rather independently from the size of \(n\).

Posterior classification performance

Given that wew have simulated items and thus know their ‘true’ values, we can also assess model performance by computing and comparing conventional classification performance metrics. Specfically, in order to assess models’ classification performance, I have first obtained items’ true classes from the simulated codings dataset. Next, I have compared model-based classifications for each value of \(n\), \(n[i]\), each chain, and each iteration to true classes. Using this data, I have then computed statistics (mean, std. dev., and 5% and 95% percentiles) of performance measures across chains and iterations for each combination of \(n\) and \(n_i\). The following figure visualizes these statistics for different classification performance metrics. (Note the scaling of the x-axis, 90% of values are in the [.7,1] range.)

• Accuracy, defined as the proportion of correctly classified items, increases (weakly) as \(n_i\) is increased. There exist no substantial accuracy differentials across values of \(n\), however. (Only 90%-CIs get tighter as \(n\) is increased, which is due to higher precision resulting from computing accuracy from a increasingly larger samples.)
• Recall (also: true-positive rate, TPR), defined as the number of true-positive classifications over the sum of true-positive and false-negative classifications (i.e., over the number of all positive items), exhibits much more variability across values of \(n_i\).
• Similarly, precision, defined as the number of true-positive classifications over the sum of all positive classification (incl. false-positives), is relatively low for low \(n_i\), but increases quiet rapidly as \(n_i\) is increased. The CIs for precision estimates are comparatively wide, however.
• In contrast, TNR, the true-negative rate defined as the number of true-negative classifications over the number of all negative items, is very high already for low values of \(n_i\), and thus we observe no substantial improvements in the negative detection rates of models as the number of judgments aggregated per item is increased. This is due to the fact that with a low positice instance prevalence, our simulated coders judge in expectation true negative items about four to five times more than positive instance. There is thus more data and hence higher precision in negative detection.
• Finally, the F1-score that combines recall and precision into one metric¹ increases significantly as \(n_i\) is increased. This is because in the denominator low Precision depresses the F1-score.

The next table reports significance levels of positive F1-score change as \(n_i\) is increased by \(l\) (columns 3–9). It reports the proportion of iterations across chains for which an increase from \(n_i\) to \(n_i+l\) judgments per item induces a decrease in the F1-score. Agian, this can be used as a significance test. The logic is simple: as we want to see F1-score increases as \(l\) is increased, the smaller the proportion with a F1-score reduction, the better.

We see that while it is not possible to assert significant F1-score imrpovments for higher-values of \(n_i\) in models fitted to small-\(n\) subsets (due to smaller sample sizes and hence less power), we have reason to be confident that increasing the number of judgments if we have collect only few thus far would lead to improvments in the models classification performance.

Another approach to visualizing posterior classification performance follows Carpenter (2008) and plots the (absolute) residual classification error for items differentiating between true-positive and true-negative items:

Again we can generally see that posterior mean estimates perform comparatively poorly in correctly classifying true positives. While most true-negative items have low absoulte residual classification error (ARCE) and are hence correctly classified (correct classification for ARCE <= .5), true-positive items are sometimes misclassified, especially for low values of \(n_i\) (some true-positive items have ARCE > .5). This reflect the generally high levels of true-negative detection ability of models accross values of \(n\) and \(n_i\), in contrast to the compartively low true-positive detection ability (recall). Yet, though misclassification occurs more often for true-positive items, the confidence in these items’ classifications is limited (most ACREs are close tot the classficiation threshold of .5). What is more, this proportion is reduced as \(n_i\) is increased, as most absolute residual values approach zero or at least get lower than .5.

Because, the above plot reports proportions across the ACRE-value range that were computed within item classes (for each \(n\)–\(n_i\) combination), the total proprotions are eclipsed. Indeed, as true items make up only between 7.6666667, 9.3333333% of all items in the respective subsets, the amount of misclassification resulting from poor true-positive detection rates is actually not too grave as the following plot shows more accurately. (Note that the x-axis of this plot depict the real-valued residual classification error, not absolute values).

In these figures, negative residuals measure how much the mean posterior class estimate (across chains an iterations) of a model deviates from the true value of 1 (0). That is, residuals in model-based classification of true-positive itmes are negative in \([-1,0)\), whereas residuals in resulting from model-based classification of true-negative itmes are always positive in \((0,1]\). Actual misclassification of true-positives (true-negatives) occurs only if the residual is smaller (greater) than -.5 (.5)—hence the two vertical dashed lines at -.5 and .5. Indeed, because for each true-positive item there are about 9 true-negatives, misclassification of true-negative items occurs more often in absolute terms, whereas in relative terms models’ true-negative detection abilities (TNR) tends to be better than their true-positive detection ability (Recall).

Comparison to majority-voting based classifications

Indeed, disagreements occur not too frequently. Alternatively, we can assess agreement by computing correlations between majority voting and model-based classifications:

## # A tibble: 1 x 15
##   sample_size   n_i n_judgments Accuracy    tp    fn    fp    tn   TPR
##         <dbl> <int>       <int>    <dbl> <int> <int> <int> <int> <dbl>
## 1         200     9         200     0.98    12     4     0   184  0.75
## # … with 6 more variables: TNR <dbl>, FPR <dbl>, FNR <dbl>,
## #   Precision <dbl>, Recall <dbl>, `F1-score` <dbl>

A stiking result of ther classification performances of majorty voting is that we get no false-positives. Hence, both the Precision and TNR for all models are perfect (false-positives factor in the denominator of both metrics). And what is more, because of the strong class imbalance, the accuracies are also very close to perfect. Indeed the F1-score is a much more informative metric in presence of such intense class imbalance.

Given that we obtain posterior classifications for each iteration and each chain of each model, we can also compute how confident we can be that the model and majority voting induce different classification performances. These differences are illustrated for F1-scores in the following plot. To obtain 90% confidence bounds, the majority-voting based F1-score point estimates have been substracted from scores induced by model-based classifications of each iteration. This yielded 3000 differences that were then aggregated into means and 90%-CIs.

This shows that while majority voting outperforms model-based classifications in terms of F1-scores for all values of \(n\) (though not significantly for all values), we see that for higher \(n_i\) the model tends to perform (significantly) better. Specifically, for \(n_i > 6\) (differences for \(n_i = 6\) may be induced by random tiebreaking in majority voting), we are 95% certain for all values of \(n\) that the model-based classification yield higher F1-scores than does majority voting. Note that the choice of comparing the F1-scores of model and majority voting induced comparisons was theoretically informed. In presence of such strong class imbalance, the accuracy is not a good performance criterion. By taking both precision and recall into account, we can achieve high F1-scores only if our aggregation method performs also reasonable well in correctly classifying true-positive items—a job the BBA model is doing better than the majority vote for \(n_i > 6\) in the simulated data.